Skip to content

fix: handle azure keyvault throttling with retry logic #237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

thomas-rabiller
Copy link

Include retry logic to deal with Azure KeyVault throttling errors that may occur during large sign batches, as described in Microsoft docs:
https://learn.microsoft.com/en-us/azure/key-vault/general/overview-throttling

@thomas-rabiller
Copy link
Author

#135

@PhonicUK
Copy link

This also looks like a solution for #249

@fabricioferreira
Copy link

I am looking forward to this PR getting approved.

Copy link

@fabricioferreira fabricioferreira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes looks good.

@fabricioferreira
Copy link

fabricioferreira commented Oct 8, 2024

Hi @vcsjones , any reason not to proceed with this merge? Azure's key vault throttling is insane and keeps failing our pipelines. Thanks!

@casuffitsharp
Copy link

I believe this issue was addressed in #251. It’s included in v6.0.0-beta1 (check the release announcement here: https://github.com/vcsjones/AzureSignTool/releases/tag/v6.0.0-beta1).

@larde
Copy link

larde commented Mar 5, 2025

We still have this issue on 6.0.1

@Lo1793
Copy link

Lo1793 commented Mar 14, 2025

Me too, retry mechanism does not seem to work.

@fedorbirjukov
Copy link

The retry mechanism is indeed part of 6.0.0. See release notes or the notes of the beta mentioned above (link: #271):

Concurrency

By far one issue that gets reported that most are failures from excessive concurrency. There are two parts that can result failures from throttling. The first is Azure. Azure limits the number of operations a KeyVault can perform in a given period. The second is timestamping from a Certificate Authority.

There are two changes in this release to help with this.

  1. The --max-degree-of-parallelism (-mdop) now defaults to 4, meaning no more than 4 sign operations will happen concurrently. To revert back to the previous behavior, set --max-degree-of-parallelism to 0, or a higher number if you still want some limit on concurrent operations.
  2. The signing operation will now retry failed signing operations if the failure was the result of Azure performing throttling.

So, it must be the CA throttling that limits the signing throughput.

This PR should be abandoned.

@bdagnin
Copy link

bdagnin commented Apr 3, 2025

For us it does seem to be the timestamp service that's causing throttling failures. The Azure retry mechanism in v6.0.1 doesn't kick in at all for these.

I've pulled v6.x into this change and created a build here: https://github.com/bdagnin/AzureSignTool/releases/download/v6.0.2-alpha.1.retry/AzureSignTool.exe
This and -mdop 1 has solved our signing failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants